Autonomous driving systems remain brittle in rare, ambiguous, and out-of-distribution scenarios, where human driver succeed through contextual reasoning. Shared autonomy has emerged as a promising approach to mitigate such failures by incorporating human input when autonomy is uncertain. However, most existing methods restrict arbitration to low-level trajectories, which represent only geometric paths and therefore fail to preserve the underlying driving intent. We propose a unified shared autonomy framework that integrates human input and autonomous planners at a higher level of abstraction. Our method leverages Vision Language Models (VLMs) to infer driver intent from multi-modal cues -- such as driver actions and environmental context -- and to synthesize coherent strategies that mediate between human and autonomous control. We first study the framework in a mock-human setting, where it achieves perfect recall alongside high accuracy and precision. A human-subject survey further shows strong alignment, with participants agreeing with arbitration outcomes in 92% of cases. Finally, evaluation on the Bench2Drive benchmark demonstrates a substantial reduction in collision rate and improvement in overall performance compared to pure autonomy. Arbitration at the level of semantic, language-based representations emerges as a design principle for shared autonomy, enabling systems to exercise common-sense reasoning and maintain continuity with human intent.

Within a few short years, a new wave of electric and self-driving vehicles will transform personal transportation as we know it. But for carmakers, these technologies will demand a staggering level of investment. They may not like the cost, but surviving through the radical new automotive era requires it. With the global economy cooling, German carmakers and suppliers seem set to embark on unprecedented collaboration, in the hope of sharing some of these development and production costs. BMW, Volkswagen and Daimler, the maker of Mercedes-Benz cars, are in talks on a formal collaboration to work on key technologies and industry standards for autonomous driving. Major parts suppliers, including Bosch, Continental and ZF, are also known to be taking part.